server: bench: minor fixes #10765
Merged
server: bench: minor fixes #10765
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- support openAI streaming standard output with [DONE]\n\n - export k6 raw results in csv - fix too many tcp idle connection in tcp_wait - add metric time to emit first token
- fix when prometheus not started - wait for server to be ready before starting bench
ngxson
approved these changes
Dec 31, 2024
There was a problem hiding this comment.
I don't have the hw to test, but LGTM.
Thought, I'm looking forward to migrate to python solution like Locust (as mentioned in the PR description). This can simplify a lot in the installation process, while giving much more flexibility for the script (Ideally, we only need single bench.py script in the future that can do all at once)
| @@ -89,6 +90,9 @@ export default function () { | |||
| ], | |||
| "model": model, | |||
| "stream": true, | |||
| "stream_options": { | |||
| "include_usage": true, // False to be supported in llama.cpp server | |||
There was a problem hiding this comment.
Not sure what you mean here, but in llama.cpp we ignore include_usage and always include include the usage info.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
After a nice exchange with @ngxson, this is a minor change to the current server bench framework in order to refresh it a bit. Although the target would be to replace k6/xk6-sse with something like Locust (to be assessed) , python based.
Changes
Tests (phi2 on RTX 3050)
LLAMA_SERVER_BIN_PATH=../../../cmake-build-debug/bin/llama-server python bench.py \ --runner-label local \ --name local \ --branch `git rev-parse --abbrev-ref HEAD` \ --commit `git rev-parse HEAD` \ --scenario script.js \ --duration 5m \ --hf-repo ggml-org/models \ --hf-file phi-2/ggml-model-q4_0.gguf \ --model-path-prefix models \ --parallel 4 \ -ngl 33 \ --batch-size 2048 \ --ubatch-size 256 \ --ctx-size 4096 \ --n-prompts 200 \ --max-prompt-tokens 256 \ --max-tokens 256Results: